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(54) DATA REPRODUCING DEVICE, DATA REPRODUCING METHOD, AND INFORMATION 
TERMINAL 



(57) Each data in MIDI, audio, text and images to be 
received in a data receiving section (3) is SMF-format- 
ted data including event information and event-execut- 
ing delta time. A data assorting section 4 assorts data 
on a kind-by-kind basis depending upon a delta time of 
each of received data. The assorted ones of data are 
respectively reproduced in a MIDI reproducing section 
(1 1 ), an audio reproducing section (1 2), a text reproduc- 
ing section (13) and an image reproducing section (14). 



The data reproduced in the MIDI reproducing section 
and audio reproducing section is mixed in a mixer (15) 
and outputted as sound from a speaker (19), while the 
data reproduced in the text reproducing section and im- 
age reproducing section is mixed in a mixer (16) and 
displayed as visible information on a display (20). Be- 
cause each of data is reproduced in the timing according 
to the delta time, synchronization can be easily provided 
between different kinds of data, as sound and images. 
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Description 

TECHNICAL FIELD 

[0001] The present invention relates to a data repro- 
ducing apparatus, data reproducing method and infor- 
mation terminal used in reproducing the data different 
in attribute, such as sound and images. 

BACKGROUND OF THE INVENTION 

[0002] Various ones of information are supplied 
through the network due to the development of multime- 
dia. These of information include, representatively, 
sound, text or images. For instance, in communications 
karaoke taken as an example, music titles and words 
are text information, accompaniment melodies and back 
choruses are sound information, and background mo- 
tion pictures are image information. 
[0003] In communications karaoke, these various 
kinds of information are simultaneously distributed 
through the network so that each of information is repro- 
duced on the terminal unit. By providing synchronization 
mutually between these of information, character color 
of words is varied or motion picture is varied in the 
progress of music. 

[0004] Conventionally, in order to provide synchroni- 
zation, clocks have been provided in the respective pro- 
grams for processing each of the information of sound, 
text and images, whereby the synchronizing process 
has been made according to the time information of 
clocks. Due to this, where the system load is increased 
or so, there is possible mutual disagreement between 
the clocks to thereby causing so-called synchronizing 
deviation and hence deviated timing of outputting each 
of information. Thus, there has been the occurrence of 
such troubles as disagreement in between sound and 
images. 

[0005] Meanwhile, the data of sound, text, images or 
the like is read out by accessing, every time, a file ac- 
cording to a command thus requiring time in processing. 
Furthermore, because the files have been separately 
prepared on an each-data basis, there also has been a 
problem of troublesome file management. 

DISCLOSURE OF THE INVENTION 

[0006] Therefore, it is an object of the present inven- 
tion to provide a data reproducing apparatus and data 
reproducing method capable of easily providing syn- 
chronization upon reproducing various kinds of informa- 
tion different in attribute. 

[0007] Another object of the invention is to provide a 
data reproducing apparatus easy in file management 
without the necessity of preparing files on a kind-by-kind 
basis of data. 

[0008] Another object of the invention is to provide a 
data reproducing apparatus capable of easily burying 



arbitrary information, such as sound, text and images, 
in an existing data format. 

[0009] Another object of the invention is to provide a 
data reproducing apparatus adapted for communica- 
5 tions karaoke. 

[0010] Another object of the invention is to provide a 
data reproducing apparatus capable of obtaining music 
play full in realism feeling. 

[0011] Another object of the invention is to provide a 
10 data reproducing method capable of reducing the trans- 
fer amount of data in the case of repetitively reproducing 
data. 

[0012] Another object of the invention is to provide a 
data reproducing method satisfactorily having a small 

15 capacity of a communication line. 

[0013] Another object of the invention is to provide a 
data reproducing method capable of further reducing 
the data amount of reproduced data. 
[0014] Another object of the invention is to provide a 

20 data reproducing method capable of suppressing noise 
occurrence during data reproducing. 
[0015] Another object of the invention is to provide a 
data reproducing apparatus and data reproducing meth- 
od capable of processing data at high speed. 

25 [0016] Another object of the invention is to provide a 
data reproducing apparatus capable of stably reproduc- 
ing data regardless of capacity variation on the trans- 
mission line. 

[0017] Another object of the invention is to provide an 
30 information terminal capable of downloading various 
kinds of information different in attribute, such as sound, 
text and images, and reproducing these for output as 
sound or visible information. 

[0018] Another object of the invention is to provide an 

35 information terminal capable of carrying out a proper 
process for interrupt signals in the information terminal 
having a function of a phone or game machine. 
[0019] Another object of the invention is to provide an 
information terminal capable of downloading and mak- 

40 jng use of music, words and jacket picture data on CD 
(Compact Disk) or MD (Mini Disk). 
[0020] Another object of the invention is to provide an 
information terminal capable of making use of each of 
downloaded data by storage to a small-sized informa- 

45 tion storage medium. 

[0021] Another object of the invention is to provide a 
data reproducing apparatus that, where receiving to 
view commercial information, the service by the com- 
mercial provider is available. 

50 [0022] In the present invention, MIDI is an abbrevia- 
tion of Musical Instrument Digital Interface, that is an 
International Standard for mutual communications with 
signals of music play between the electronic musical in- 
struments or between the electronic musical instrument 

55 and the computer. Meanwhile, SMF is an abbreviation 
of Standard MIDI File, that is a standard file format com- 
prising time information called delta time and event in- 
formation representative of music play content or the 
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like. It should be noted that the terms "MIDI" and "SMF" 
in the present Specification are used in the above mean- 
ings. 

[0023] In the invention, the data to be received in- 
cludes event information and time information an event 
is to be executed, and comprises data in a format of SM F 
or the like. The received data is assorted on a kind-by- 
kind basis depending upon respective ones of time in- 
formation so that an event of assorted data is executed 
to reproduce the data. 

[0024] In the invention, because the time information 
and the information of sound, text, images and the like 
are integral, the time information can be utilized as syn- 
chronizing information by reproducing various kinds of 
data according to the time information possessed by 
them. As a result, it is possible to easily provide syn- 
chronization to between different kinds of data as in 
sound and images. Also, there is no need to separately 
prepare and manage files on a kind-by-kind basis of da- 
ta, thus facilitating file management. Furthermore, there 
is no necessity to access various kinds of files every 
time, thereby increasing the speed of processing. 
[0025] The reception data can be configured with first 
data having MIDI event information and second data 
having event information ofotherthan MIDI. The second 
data can be considered, for example, the data concern- 
ing text, images, audio or the like. 
[0026] The MIDI event is a gathering of the com- 
mands for controlling tone generation of the musical in- 
strument. For example, instruction command forms are 
taken as "Start Tone Generation of Do" "Stop Tone Gen- 
eration of Do". Also, the MIDI event is to be added with 
a delta time as time information and turned into an SMF- 
formatted data so that such an event as "Tone Genera- 
tion Start of Do" "Tone Generation Stop of Do" is exe- 
cuted when a predetermined time comes according to 
a time represented by the delta time. 
[0027] On the other hand, the other events than MIDI 
include a META event or a system exclusive event. 
These events can be extended in format as hereinafter 
referred so that various kinds of data can be buried in 
the extended format. The use of such an SMF extended 
format can easily record various kinds of data, such as 
sound and images without major modification to the for- 
mat. 

[0028] The present invention receives the data having 
each of event information of MIDI, text and images to 
output reproduced MIDI data as sound and reproduced 
text and image data as visible information, thereby mak- 
ing possible to realize a data reproducing apparatus 
suited for karaoke. In this case, the addition of voice be- 
sides MIDI as audio makes it possible to reproduce a 
musical-instrument playing part in MIDI and a vocal part 
such as back chorus in voice, thus realizing music play 
full in realism feeling. 

[0029] In the case of repetitively reproducing the sec- 
ond data having event information other than MIDI, it is 
preferred to previously store first-received data to a 



memory so that, when repetitively reproduce the data, 
only the time information concerning reproducing is 
transmitted by the second data. By doing so, the transfer 
amount of data can be decreased. 
5 [0030] Also, in the case of reproducing the second da- 
ta following the first data, it is preferred to divide the re- 
produced data in the second data into a plurality of data, 
to transmit a data group having the plurality of divisional 
data inserted between the preceding ones of first data, 
10 to extract the inserted divisional data from the data 
group at the reception side, and to combine the extract- 
ed divisional data into reproduced data. This smoothen 
the amount of transmission data and satisfactorily 
makes the communication line with small capacity. In 
15 this case, the extracted divisional data is sequentially 
stored to the memory in a chronological fashion and the 
area in which divisional data is stored is recorded by a 
start address of the following divisional data to which the 
relevant divisional data is coupled, thereby easily, pos- 
20 itively combining together the divisional ones of data. 
[0031] Furthermore, by cutting away the silence sec- 
tions of the reproduced data to be recorded to the sec- 
ond data, the data amount can be reduced furthermore. 
In this case, it is preferred to carry out a fade-in/out proe- 
ms ess to a signal in the vicinity of a rise portion and fall 
portion of the reproduced data in view of suppressing 
noise occurrence. 

[0032] In another embodiment of a data reproducing 
apparatus of the invention, each of data different in at- 

30 tribute is assorted, for each unit section, and stored to 
a storage section depending upon the time information 
thereof, sequentially being read out of the storage sec- 
tion and reproduced in the next unit section. With this, 
because the process of received data is made in a pipe- 

35 line form, processing is possible at higher speed. Also, 
time synchronization can be easily provided by manag- 
ing data time information and unit-section time width to 
forward only the data to be processed in the relevant 
unit section to the storage section. 

40 [0033] The data reproducing apparatus according to 
the invention can adopt a stream scheme that data is 
reproduced while being downloaded. In this case, if the 
amount of data being consumed by reproducing ex- 
ceeds the amount of data being taken, there arise data 

45 insufficiency and discontinuity in sound, images or the 
like. Accordingly, by caching data to a required amount 
to thereafter start reproducing, data can be continuously 
reproduced without encountering discontinuity. 
[0034] The data reproducing apparatus according to 

50 the invention can be mounted on an information terminal 
such as cellular phone or game machine, wherein vari- 
ous kinds of data can be downloaded from a server by 
making use of a terminal communicating function. By 
providing a speaker for outputting sound or display for 

55 displaying text and images to the information terminal, 
music or images can be heard or viewed on the terminal. 
In the case of a phone, it is preferred to output an in- 
coming tone by prohibiting sound from being outputted 
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from the speaker upon receiving an incoming signal. In 
the case of a game machine, an effect sound due to 
MIDI can also be outputted together with sound from the 
speaker. 

[0035] A small-sized information storage medium can 
be removably provided on the data reproducing appa- 
ratus according to the invention, whereby various kinds 
of downloaded data can be stored, for reutilization, in 
the information storage medium. For example, if down- 
loading music data in MIDI or audio, music words or 
commentary data in text and jacket picture data in im- 
ages respectively, the information storage medium itself 
can be utilized as CD or MD. 

[0036] In the invention, by previously including an In- 
ternet URL and the information concerning service to be 
offered in the URL in commercial-information text data 
to be received and then causing jump to a homepage of 
the URL following the reproducing of commercial, vari- 
ous services can be offered to the commercial viewers. 



a silent section. 

Fig. 18 is a flowchart showing a process of silence 
sections. 

Fig. 19 is a blockdiagram showing another example 
5 of a data reproducing apparatus of the invention. 

Fig. 20 is a flowchart showing another example of 
a data reproducing method of the invention. 
Fig. 21 is a figure explaining the principle of time 
operation in data assortment. 
10 Fig. 22 is a flowchart showing a procedure of data 
assortment. 

Fig. 23 is a flowchart showing the operation of each 
data reproducing section. 
Fig. 24 is a time chart of a data process overall. 
Fig. 25 is a figure explaining the operation of data 
reception in a stream scheme. 
Fig. 26 is a time chart of data reception. 
Fig. 27 is a time chart explaining data cache. 
Fig. 28 is a blockdiagram showing another example 
of a data reproducing apparatus of the invention. 
Fig. 29 is a time chart showing the operation of the 
apparatus of Fig. 28. 

Fig. 30 is a blockdiagram showing another example 
of a data reproducing apparatus of the invention. 
Fig. 31 is a time chart showing the operation of the 
apparatus of Fig. 30. 

Fig. 32 is a flowchart in the case of implementing a 
charge discount process using the data reproduc- 
ing apparatus of the invention. 
Fig. 33 is a figure chronologically showing each of 
data configuring CM. 

Fig. 34 is an example of a tag to be added to text 
data. 

Fig. 35 is a flowchart in the case of implementing 
the service with an available period using the data 
reproducing apparatus of the invention. 
Fig. 36 is an example of a tag to be added to text 
data. 

Fig. 37 is a figure showing a cellular phone mounted 
with the data reproducing apparatus of the inven- 
tion. 

Fig. 38 is a table figure of a memory built in an in- 
formation storage medium. 

Fig. 39 is a figure showing a system using a cellular 
phone. 

BEST MODE FOR CARRYING OUT THE INVENTION 

[0038] An example of a data reproducing apparatus 
is shown in Fig. 1 . In Fig. 1 , 1 a, 1 b is a file on which data 
is recorded, wherein 1a is for example a file existing in 
a server on the Internet, and 1 b is for example a file on 
a hard disk within the apparatus. 
[0039] 2 is a CPU for controlling the data reproducing 
apparatus overall, and structured including a data re- 
ceiving section 3 and a data assorting section 4. Al- 
though besides these the CPU 2 includes blocks having 
various functions, those in the present invention are not 



20 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0037] 

Fig. 1 is a block diagram showing an example of a 25 
data reproducing apparatus of the present inven- 
tion. 

Fig. 2 is a figure showing a format of SM F-formatted 
reception data. 

Fig. 3 is a format example of the data concerning 30 
MIDI. 

Fig. 4 is a format example of the data concerning 
simplified-formed MIDI. 

Fig. 5 is a format example of the data concerning 
audio, text and images. 35 
Fig. 6 is a format example of a META event con- 
cerning control. 

Fig. 7 is another format example of the data con- 
cerning audio, text and images. 

Fig. 8 is a format example of a data row. 40 
Fig. 9 is a flowchart showing an example of a data 
reproducing method according to the invention. 
Fig. 10 is a flowchart showing another example of 
a data reproducing method according to the inven- 
tion. 45 
Fig. 11 is a figure explaining a repetitive reproducing 
process of data. 

Fig. 12 is a flowchart of the repetitive reproducing 
process. 

Fig. 1 3 is a figure showing the principle of previous 50 
forwarding of data. 

Fig. 14 is a figure showing an insertion example of 
divisional data. 

Fig. 1 5 is a figure showing the content of a memory 
storing divisional data. 55 
Fig. 1 6 is a flowchart in the case of storing divisional 
data to a memory. 

Fig. 1 7 is a waveform diagram of audio data having 
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in direct relation and hence omittedly shown. The data 
receiving section 3 has access to the file 1a, 1b to re- 
ceive the data stored therein. The data in the file 1a is 
received, through a wire or wirelessly. These of received 
data are temporarily stored to a buffer 3a. The data as- 
sorting section 4 assorts the data received by the data 
receiving section 3, into a data reproducing section 6 on 
a kind-by-kind basis. 

[0040] The data reproducing section 6 is configured 
with a MIDI reproducing section 11 to reproduce the data 
concerning MIDI, an audio reproducing section 12 to re- 
produce the data concerning audio, a text reproducing 
section 13 concerning a text and an image reproducing 
section 14 to reproduce the data concerning an image. 
The MIDI reproducing section 11 has a sound-source 
ROM 1 1 a storing the sound-source data of various mu- 
sical instruments used for the music to be reproduced. 
This sound-source ROM 11a can be replaced with a 
RAM and mounted with the built-in data replaced. The 
image reproducing section 14 has a function to repro- 
duce still images and motion images. 
[0041] 15 is a mixer for mixing the outputs of the MIDI 
reproducing section 11 and audio reproducing section 
12, and 16 is a mixer for mixing the outputs of the text 
reproducing section 13 and image reproducing section 
14. The mixer 15 is provided with a sound effect section 
15a to process for echo addition or the like, while the 
mixer 1 6 is provided with a visual effect section 16a to 
process a special effect addition to an image. 17 is an 
output buffer for temporarily storing the output of the 
mixer 1 5, and 1 8 is an output buffer for temporarily stor- 
ing the output of the mixer 1 6. 1 9 is a speaker as a sound 
generating section for outputting sound depending on 
the data of the output buffer 1 7, and 20 is a display for 
displaying visible information of characters, illustrations 
or the like on the basis of the data of the output buffer 1 8. 
[0042] To the data receiving section 3 is inputted the 
SMF-formatted data recorded in the files 1a, 1b is. The 
SMF-formatted data generally comprises time informa- 
tion, called delta time, and event information represent- 
ative of a music-play content or the like, and includes 
three types shown in Figs. 2(a) - (c) in accordance with 
a kind of event information, (a) is the data having the 
event information comprising a MIDI event, (b) is the da- 
ta having the event information comprising a META 
event and (c) is the data having event information com- 
prising a Sys. Ex event. 

[0043] The detail of the MIDI event is shown in Fig. 3. 
Fig. 3(a) is the same as Fig. 2(a). The MIDI event com- 
prises status information and data, as shown in Figs. 3 
(b) and (c). Fig. 3(b) is an event of a tone-generation 
start command wherein the status information is record- 
ed with a musical instrument kind, data 1 is with a scale 
and data 2 is with a tone intensity, respectively. Mean- 
while, Fig. 3(c) is an event of a tone-generation stop 
command, wherein the status information is recorded 
with a musical instrument kind, data 3 is with a scale 
and data 4 is with a tone intensity, respectively. In this 



manner, the MIDI event is an event storing music-play 
information wherein one event configures a command, 
e.g. "Generate Tone Do With Piano Sound At this Inten- 
sity". 

5 [0044] Fig. 4 shows an example of a simplified MIDI 
format reduced in data amount by simplifying the format 
of Fig. 3. Although Fig. 3 separately configures the tone 
start command and the tone stop command, whereas 
Fig. 4 integrates tone generation and stop into one by 

10 adding a tone-generating time in the data. Meanwhile, 
tone intensity data is omitted and scale data is included 
in the status information. Incidentally, although the for- 
mat of Fig. 4 is not a standard format alike an SMF, the 
data to be dealt with in the invention includes the other 

15 formats than the one alike an SMF. 

[0045] The detail of the META event is shown in Fig. 
5. Fig. 5(a) is the same as Fig. 2(b). The MEAT event, 
an event for transferring data or controlling for repro- 
duce start/stop, is allowed to extend the format so that 

20 a variety of data can be buried within the extended for- 
mat. Figs. 5(b) - (e) show format examples of extended 
META events, (b) is a format buried with audio data, (c) 
a format buried with text data, (d) a format buried with 
image data and (e) a format buried with text data and 

25 image data, respectively. The image includes a motion 
image, besides a still image such as illustration and pic- 
ture. 

[0046] The FFh at the top is a header showing that 
this event is a META event. The next 30h, 31 h, ... are 

30 identifiers representing that the format f the M ETA event 
is an extended format. Meanwhile, len represents a data 
length of the META event, type a format of data to trans- 
fer, and id a data number, respectively. Event is to show 
an event content to be executed, and represented by a 

35 command, e.g. "Start Audio Data Transfer" or "End Im- 
age Data Transfer". The end point of such data can be 
known from a len value representative of a data length. 
[0047] The META event includes a format concerning 
control besides the extended format recording the data 

40 as above. Fig. 6 is one example of the same, wherein 
(a) shows an event format for reproduce start and (b) 
for reproduce stop. 1 0h in (a) and 1 1 h in (b) are, respec- 
tively, commands for reproduce start and reproduce 
stop. The other FFh, len, type and id are the same as 

45 Fig. 5, hence explanations being omitted. 

[0048] The detail of the Sys. Ex event is shown in Fig. 
7. Fig. 7(a) is the same as Fig. 2(c). The Sys. Ex event 
is called an exclusive event, e.g. an event concerning 
the set information for setting in a system adapted for 

50 the orchestra and the like. This Sys. Ex event is also 
possible to extend so that various kinds of data can be 
buried in the extended format. Figs. 7(b)-(e) show a for- 
mat example of an extended Sys. Ex event, which is in 
a format similar to Fig. 5. 

55 [0049] The SMF-formatted data is configured as 
above, these of which data are combined in numbers to 
constitute a series of data. Fig. 8 shows an example of 
such a data row. M is data concerning MIDI and has a 
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format shown in Fig. 3. A is data concerning audio and 
has a format shown in Fig. 5(b). T is data concerning 
text and has a format shown in Fig. 5(c). P is data con- 
cerning an image and has a format shown in Fig. 5(d). 
Incidentally, the order of arrangement of each of data is 
not limited to Fig. 8 but there can exist a variety of pat- 
terns. Also, although in Fig. 8 the data of audio, text and 
images are described in a META event, these can be 
recorded in the Sys. Ex event. Each of data M, A, T, P 
is configured as a packet, these of which are chained 
into a series of data row. The data row is to be received 
by a data receiving section 3 of Fig. 1 and stored in the 
buffer 3a. 

[0050] The received data is assorted in the data as- 
sorting section on the basis of a delta time AT thereof, 
to execute an event in the data reproducing section 6 
and reproduce the data. The timing an event is to be 
executed is determined by the delta time AT. Namely, 
an event is executed when the relationship between a 
lapse time AT from the immediately preceding event ex- 
ecuted and a delta time AT of an event to be currently 
executed is in At > AT. Namely, if a certain event is ex- 
ecuted, the lapse time from a start of the event is count- 
ed. When the lapse time is equal to a delta time of the 
next event or exceeds it (there is a case not exactly co- 
incident with but exceeding a delta time because the 
time resolving power by the CPU is finite), the next event 
is executed. In this manner, the delta time is information 
representing to execute the current event depending on 
what amount of time elapses from the immediately pre- 
ceding event. Although the delta time does not repre- 
sent an absolute time, the time from a start of reproduc- 
ing can be calculated by integrating the delta time. 
[0051] Hereunder, explanation is made on the detail 
of reproducing in each section of the data reproducing 
section 6. First, explained is the operation of reproduc- 
ing in the MIDI reproducing section 11. In Fig. 1, the data 
assorting section 4 of the CPU 2 sequentially reads the 
received data out of the buffer 3a according to a program 
stored in a not-shown ROM. If the read-out data is data 
M concerning MIDI (Fig. 3), its event information is sup- 
plied to the MIDI reproducing section 11. If the event 
content is, for example, a command "Generate Tone Mi 
With Piano Sound", the MIDI reproducing section 11 de- 
crypts this command to read a piano sound from the 
sound-source ROM 11a and creates a synthesizer 
sound by a software synthesizer thereby starting to gen- 
erate a tone at a scale of mi. From then on, the CPU 2 
counts a lapse time. If this lapse time is equal to the 
delta time attached to the next event "Stop Tone Gen- 
eration of Mi" or exceeds it, this command is supplied 
to the MIDI reproducing section 11 so that the MIDI re- 
producing section 11 decrypts the command to cease 
the tone generation of mi. In this manner, the tone of mi 
is reproduced with a piano sound only in the duration of 
from a tone generation start to a tone generation stop. 
[0052] Next, the CPU 2 counts a lapse time from a 
tone generation stop of mi. If this lapse time is equal to, 



for example, a delta time attached to the next event 
"Generate Tone Ra With Piano Sound" or exceeds it, 
this command is supplied to the MIDI reproducing sec- 
tion 11. The MIDI reproducing section 11 decrypts this 

5 command to read a piano sound from the sound-source 
ROM 1 1 a and create a synthesizer sound thereby start- 
ing tone generation at a scale of ra. Then, from then on 
the CPU 2 counts a lapse time. If the lapse time is equal 
to a delta time attached to the next event "Stop Tone 

10 Generation of Ra" or exceeds it, this command is sup- 
plied to the MIDI reproducing section 11. The MIDI re- 
producing section 11 decrypts this command to stop the 
tone generation of ra. In this manner, the tone of ra is 
reproduced with a piano sound only in the duration of 

15 from the tone generation start to the tone generation 
stop. By the repetition of such an operation, the MIDI 
reproducing section 11 reproduce a sound according to 
MIDI. 

[0053] Next, explanation is made on reproducing of 

20 the data having other event information than MIDI. As 
in the foregoing, each of data of an audio, text and image 
is recorded in a META event (Fig. 5) or Sys. Ex event 
(Fig. 7). In Fig. 1 , the data assorting section 4 sequen- 
tially reads the received data from the buffer 3a, similarly 

25 to the above. In the case that the read-out data is data 
A concerning audio, the event information of the read- 
out data is assorted to the audio reproducing section 1 2 
according to a delta time. The audio reproducing section 
12 decrypts the content of the relevant event and exe- 

30 cutes the event, thus reproducing an audio. Where the 
read-out data is data T concerning a text, the event in- 
formation of the read-out data is assorted to the text re- 
producing section 1 3 according to a delta time. The text 
reproducing section 13 decrypts the content of the rel- 

35 evant event and executes the event, thus reproducing 
a text. Where the read-out data is data P concerning an 
image, the event information of the read-out data is as- 
sorted to the image reproducing section 1 4 according to 
a delta time. The image reproducing section 1 4 decrypts 

40 the content of the relevant event and executes the 
event, thus reproducing an image. 
[0054] More specifically, if the audio reproducing sec- 
tion 12 receives, for example, an event "Generate 
Sound B" from the data assorting section 4, the audio 

45 reproducing section 1 2 decodes and reproduces the da- 
ta of a sound B added to the relevant event. From then 
on, the CPU 2 counts a lapse time. If the lapse time is 
equal to, for example, a delta time attached to the next 
event "Display Character C" or exceeds it, the text re- 

50 producing section 13 decodes and reproduces the data 
of a character C added to the relevant event. Next, the 
CPU 2 counts a lapse time from the reproducing of the 
character C. If the lapse time is equal to, for example, a 
delta time attached to the next event "Display Illustration 

55 D" or exceeds it, the image reproducing section 14 de- 
codes and reproduces the data of an illustration D added 
to the relevant event. This, in this respect, is basically 
similar to the principle of the reproducing of MIDI data. 
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[0055] The above explanation was made, for purpose 
of convenience, separately between the reproduce op- 
eration by the MIDI reproducing section 11 and the re- 
produce operation by the other reproducing section 12 
- 14 than MIDI. However, actually as shown also in Fig. 
8 the data receiving section 3 is chronologically, mixedly 
inputted by data M having a MIDI event and data A, T, 
P having an event other than MIDI. For example, differ- 
ent kinds of data is sequentially inputted, e.g. MIDI (M) 
-> illustration (P) -> text (T) -> MIDI (M) -> audio (A) -> 
motion image (P) -» .... The data assorting section 4 as- 
sorts these of data to each reproducing section 11-14 
on a kind-by-kind basis according to a delta time. Each 
reproducing section 11-14 carries out a reproducing 
process of data corresponding thereto. 
[0056] The data reproduced in the MIDI reproducing 
section 11 and the data reproduced in the audio repro- 
ducing section 1 2 are mixed together by a mixer 1 5 and 
echo-processed by the sound effect section 15a, and 
thereafter temporarily stored in the output buffer 1 7 and 
outputted as a sound from the speaker 1 9. On the other 
hand, the data reproduced in the text reproducing sec- 
tion 13 and the data reproduced in the image reproduc- 
ing section 1 4 are mixed together by a mixer 1 6 and sub- 
jected to a special image effect or the like in the visual 
effect section 15a, and thereafter temporarily stored in 
the output buffer 1 8 and displayed as visible information 
on the display 20. Then, when the data assorting section 
4 receives a META event for reproduce stop shown in 
Fig. 6(b), the reproducing of data is ended. 
[0057] In this manner, in the data reproducing appa- 
ratus of Fig. 1 , each of data can be reproduced through 
assortment on a kind-by-kind basis from a data row 
mixed with MIDI, audio, text and images. Upon repro- 
ducing a textor image, adeltatime is referred to similarly 
to MIDI reproducing so that data is reproduced in the 
timing dependent upon the delta time. Consequently, by 
merely describing a delta time, synchronization can be 
easily made between different kinds of data, such as 
sound and images. Meanwhile, because there is no 
need to build clocks in a program for processing each 
of data as required in the conventional, there occur no 
problems with synchronization deviation due to the mis- 
match between the clocks. 

[0058] Fig. 9 is a flowchart showing a data reproduc- 
ing method in the reproducing apparatus of Fig. 1 , show- 
ing a procedure to be executed by the CPU 2. Hereun- 
der, explanation is made on the operation with exempli- 
fication that the reproducing apparatus is a reproducing 
apparatus for communications karaoke. Incidentally, the 
step of flowchart will be abbreviated "S". 
[0059] If the data receiving section 3 receives data 
from a file 1a of a server on a network through a com- 
munication line (S101), this received data is stored to 
the buffer 3a (S1 02). Next, the data assorting section 4 
reads out the data of the buffer 3a and counts a lapse 
time from a execution of the immediately preceding 
event (S1 03). Then, determination is made as to wheth- 



er this lapse time coincides with a time represented by 
a delta time (or exceeds it) (S104). If the delta time is 
not exceeded (S104 NO), S103 is returned to continue 
counting of the lapse time. If the lapse time coincides 
5 with or exceeds the delta time (S104 YES), a process 
of data is entered. 

[0060] In the process of data, a kind of the received 
data is first determined. Namely, it is determined wheth- 
er the received data is MIDI data M or not (S105). If it is 

10 a MIDI data (S105 YES), this is assorted to the MIDI 
reproducing section 11 so that a synthesizer sound is 
created in the MIDI reproducing section 11 (S111). The 
detailed principle of the same was already described, 
hence the explanation being omitted herein. Due to 

15 sound reproducing by the synthesizer, a karaoke ac- 
companying melody is outputted from the speaker 19. 
[0061] If the received data is not MIDI data M (S105 
NO), then whether audio data A or not is determined 

(51 06) . If it is audio data A (S1 06 YES), this is assorted 
20 to the audio reproducing section 12 so that an audio 

process is carried out in the audio reproducing section 
12 thereby reproducing an audio (S112). The detailed 
principle of the same was already described, the expla- 
nation being omitted herein. By reproducing of an audio, 
25 a vocal such as a background chorus is outputted from 
the speaker 1 9. 

[0062] If the received data is not audio data A (S1 06 
NO), then whether text data T or not is determined 

(51 07) . If text data T (S1 07 YES), this is assorted to the 
30 text reproducing section 1 3 so that a text process is car- 
ried out in the text reproducing section 13 thereby re- 
producing a text (S113). By reproducing of a text, the 
title or words of karaoke music is displayed on the dis- 
play 20. 

35 [0063] If the received data is not text data T (S107 
NO), then whether image data P or not is determined 

(5108) . If image data P (S108 YES), this is assorted to 
the image reproducing section 14 so that a process for 
a still image or motion image is carried out in the image 

40 reproducing section 14 thereby reproducing an image 
(S114). By reproducing of an image, a background im- 
age such as an animation or motion image is displayed 
on the display 20. 

[0064] If the received data is not image data (S108 
45 NO), the same data is for example data concerning set- 
ting or control and a predetermined process in accord- 
ance with its content is carried out (S1 09). Subsequent- 
ly, it is determined whether to stop the reproducing or 
not, i.e. whether received a META event of Fig. 6(b) or 
50 not (S110). In the case the reproducing is not stopped 
(S110 NO), S101 is returned to wait for receiving the 
next data. Where stopping the reproducing (S1 1 0 YES), 
the operation is ended. 

[0065] As in the foregoing, the data reproducing ap- 
55 paratus of Fig. 1 is made as an apparatus adapted for 
communications karaoke by the provision of the sound 
reproducing section comprising the MIDI reproducing 
section 1 1 and the audio reproducing section 1 2 and the 
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visual information reproducing section comprising the 
text reproducing section 13 and the image reproducing 
section 14. In the invention, the audio reproducing sec- 
tion 12 is not necessarily required but can be omitted. 
However, by providing the audio reproducing section 12 
to reproduce a musical instrument part in the MIDI re- 
producing section 11 and reproduce a vocal part in the 
audio reproducing section 12, the vocal part can be re- 
produced with an inherent sound thus realizing a music 
play with high a realism feeling. 

[0066] Incidentally, the SMF-f or matted data to be re- 
ceived by the data receiving section 3 is stored in the 
file 1 a of the server on the network as described before. 
New-piece data is periodically uploaded to the file 1a, 
to update the content of the file 1a. 
[0067] Fig. 10 is a flowchart showing a reproducing 
method in the case where the data reproducing appa- 
ratus of Fig. 1 is used in the broadcast of television CM 
(commercial), showing a procedure to be executed by 
the CPU 2. In the figure, S121 - S124 correspond, re- 
spectively, to S101 - 104 of Fig. 9, wherein the opera- 
tions thereof are similar to the case of Fig. 9 and hence 
explanation is omitted. 

[0068] If a predetermined time is reached to enter a 
process (S124 YES), it is determined whether the re- 
ceived data is data of a music to be played in a back- 
ground of CM or not (S125). Herein, the data for a back 
music is configured by MIDI. If background music data 
(S125 YES), it is assorted to the MIDI reproducing sec- 
tion 11 to carry out a synthesizer process, thereby re- 
producing a sound (S132). This outputs a CM back- 
ground music from the speaker 19. 
[0069] If the received data is not background music 
data (S125 NO), it is then determined whether it is data 
for an announcement that an announcer is to talk or not 

(51 26) . The announce data is configured by audio data. 
If it is announce data (S126 YES), it is assorted to the 
audio reproducing section 1 2 to carry out an audio proc- 
ess thereby reproducing an audio (S133). By reproduc- 
ing of an audio, an announcer commentary or the like is 
outputted from the speaker 19. 

[0070] If the received data is not announce data (S1 26 
NO), it is then determined whether the data is text data 
representative of a product name or the like or not 

(5127) . If text data (S127 YES), this is assorted to the 
text reproducing section 13 so that a text is reproduced 
in the text reproducing section 13 and displayer on the 
display 20 (S134). 

[0071 ] If the received data is not text data (S1 27 NO), 
it is then determined whether the data is illustration data 
or not (S128). If illustration data (S128 YES), this is as- 
sorted to the illustration reproducing section 14 so that 
a still image process is carried out in the image repro- 
ducing section 14 thereby reproducing an illustration 
and displaying it on the display 20 (S135). 
[0072] If the received data is not illustration data 
(S1 28 NO), it is then determined whether the data is mo- 
tion image data or not (S129). If motion image data 



(S129 YES), this is assorted to the image reproducing 
section 14 so that a motion image process is carried out 
in the image reproducing section 14 thereby reproduc- 
ing a motion image and displaying it on the display 20 
5 (S136). 

[0073] If the received data is not motion image data 
(S129 NO), advancement is to S130. S130 and S131 
correspond, respectively, to S1 09 and S11 0 of Fig. 9 and 
the operations thereof are similar to Fig. 9, hence expla- 

10 nation being omitted. 

[0074] In the meanwhile, in the foregoing reproducing 
method, in reproducing audio, text and image data bur- 
ied in the SMF-formatted data, there are cases of repro- 
ducing the same data with repetition in a certain number 

15 of times. For example, there are cases that karaoke 
background chorus is repeated thrice or the same text 
are displayed twice in the start and end portions of CM. 
In such cases, there is a problem of increase in data 
amount if the data in the number corresponding to the 

20 number of times of repetition is buried in the format of 
Fig. 5 or Fig. 7. 

[0075] Accordingly, the method shown in Fig. 11 is to 
be considered as a solution to that. Namely, where re- 
producing the same data R with thrice repetition in the 

25 timing of t1, t2 and t3 as in (a), the transmitting side 
(server) firstly forwards once a packet buried with the 
data R as in (b) . The receiving side (data reproducing 
apparatus) stores the data R in a memory (not shown). 
During repetitive reproducing, the transmitting side for- 

30 wards only a message "Reproduced data Upon Lapse 
of Time Represented by Delta Time" without forwarding 
the data R. According to the message, when a prede- 
termined time comes according to the delta time, the re- 
ceiving side reads the data R out of the memory and 

35 reproduces it. By carrying out this operation thrice at t1 , 
t2 and t3, the data amount to be transmitted is satisfac- 
torily reduced to one-third. 

[0076] Incidentally, although exemplified herein was 
the case that the transmission data after once stored in 

40 a memory was reproduced, the method of Fig. 11 can 
be applied to the data reception of so-called a stream 
scheme that data is reproduced while being download- 
ed. In this case, the data R forwarded at t1 as a first 
reproducing time point will be stored in the memory. 

45 [0077] Fig. 12 is a flowchart showing a repetitive re- 
producing process described above, which is a detailed 
procedure in S112, S113 or S114 of Fig. 9 or a detailed 
procedure in S1 33, S1 34, S1 35 or S1 36 of Fig. 1 0. First, 
whether the received data is data R to be repetitively 

50 reproduced or not is determined (S141), wherein if not 
repetitive data (S1 41 NO), processing is made as usual 
data. If it is repetitive data (S141 YES), the number of 
times of repetition is set to a counter N in the CPU (S142) 
and the data R is read out of the memory (S1 43) to out- 

55 put it (S144). Next, the counter N is subtracted by 1 
thereby updating it into N-1 (S145). Then, whether the 
counter N has become 0 or not is determined (S146). If 
notO (S146 NO), movement is to S1 10 of Fig. 9 orS131 
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of Fig. 10. If the counter N has become 0 (S146 YES), 
the data R recorded is erased to release the memory 
(S147). 

[0078] Fig. 13 is a figure showing a principle of previ- 
ously forwarding of data in a stream scheme. In the case 
of forwarding audio or image data following MIDI data, 
as shown in (a) the amount of data is less in the MIDI 
portion whereas the amount of data abruptly increases 
in the portion of audio or image data X. (The reason of 
less data amount of MIDI is that MIDI is not tone data 
itself but acommand for controlling to generate tone and 
configured by binary data). Consequently, the data X if 
directly forwarded requires a great capacity of a com- 
munication line. 

[0079] Accordingly, as shown in Fig. 13(b) the data X 
is appropriately divided to attach the IDs of X1 , X2 and 
X3 to the divided data. These divisional data is inserted 
in the preceding MIDI data and previously forwarded, 
thereby making it possible to smoothen the amount of 
transmission data and reduce the capacity on the line. 
Although the example of partly dividing the data X was 
shown herein, the data may be divided throughout the 
entire section. 

[0080] The data following the MIDI may be a plurality 
of coexisting data X, Y as shown in Fig. 14(a). In also 
this case, the divisional data of the data X and data Y is 
each given an ID of X1 , X2, ... and Y1 , Y2, ... on an each- 
group basis. Fig. 1 4 (b) shows an example that division- 
al ones of data are inserted between the preceding ones 
of MIDI data. If a data group having divisional data thus 
inserted therein is received by the data receiving section 
3, the divisional ones of inserted data are extracted from 
the data group. By combining the extracted ones of di- 
visional data, the former reproduced data is restored. 
The detail is explained with reference to Fig. 1 5 and Fig. 
16. 

[0081 ] The received divisional data is separated from 
the MIDI data and sequentially stored to the memory in 
the chronological order of from the top data of Fig. 14 
(b). The content of the memory is shown in Fig. 15. In 
an area of each divisional data stored, a start address 
of a following one of divisional data to be coupled to the 
relevant divisional data is recorded on the each-group 
basis of X and Y. For example, a start address of data 
X2 is recorded in the end of data X1 , and a start address 
of data X3 is recorded in the end of data X2. Also, a start 
address of data Y2 is recorded in the end of data Y1 , 
and a start address of data Y3 is recorded in the end of 
data Y2. 

[0082] Fig. 1 6 is a flowchart showing the operation to 
extract divisional data and store it to the memory in the 
case where the data receiving section 3 receives a data 
group of Fig. 14(b). First, the top data X1 is read in 

(5151) , and the read data X1 is written to the memory 

(51 52) . Subsequently, data X2 is read in (S1 53), where- 
upon a start address of an area to store the data X2 is 
written to the end of the data X1 (S154) and then the 
data X2 is written to the memory (S155). Next, after 



processing the MIDI data (S156), data Y1 is read in 

(5157) and the read data Y1 is written to the memory 

(5 1 58) . Thereafter, data X3 is read in (S 1 59) whereupon 
a start address of an area to store the data X3 is written 

5 to the end of the data X2 (S1 60) and then the data X3 
is written to the memory (S1 61 ). Subsequently, data Y2 
is read in (S1 62), whereupon a start address of an area 
to store the data Y2 is written to the end of the data Y1 
(S163) and then the data Y2 is written to the memory 

10 (S164). From now on, data X4 to data X6 are similarly 
written to the memory. 

[0083] By thus recording the start addresses of the fol- 
lowing ones of divisional data to the ends of the division- 
al data stored in the memory, the divisional ones of data 

15 can be easily combined for restoration. Namely, con- 
cerning the data X, the divisional ones of data X1 , X2, ... 
X6 are coupled in a chain form through the start ad- 
dresses. Accordingly, even if the divisional data of the 
data X and the divisional data of the data Y are mixedly 

20 stored as in Fig. 15, the dataXI, X2, ... X6 are read out 
with reference to the start addresses and combined to- 
gether, the former data X can be easily restored. This is 
true for the data Y. 

[0084] Fig. 17 is a figure explaining a process of the 

25 audio data having a silent section. For example, consid- 
eration is made on the case that the voice of an an- 
nouncer is recorded as an audio signal and buried in an 
SMF format of Fig. 5(b) or Fig. 7(b). There possibly are 
pauses in the midway of announcer's voice. The data in 

30 the pause section (silent section) is data unnecessary 
in nature. Accordingly, if the silent sections of data are 
cut to bury only the required portions in the SMF format, 
the amount of data can be reduced. 
[0085] In the audio signal of Fig. 17, a section T is a 

35 silent section. Although the silent section T is a section 
having inherently a signal level of 0, actually the level is 
not necessarily limited to 0 because of mixing-in of noise 
or the like. Accordingly, a level value L in a constant 
range is fixed so that, where a section that the signal 

40 level does not exceed L continues a constant section, 
this section is given as a silent section T to create audio 
data the silent sections T are cut away. If this is buried 
in the SMF format of Fig. 5(b) or Fig. 7(b) and repro- 
duced according to the foregoing reproducing method, 

45 the amount of transmission data is satisfactorily less and 
the memory capacity on the receiving side can be econ- 
omized. 

[0086] However, by merely cutting away of the silent 
sections T, during reproducing the signal sharply rises 

50 and falls thereby causing noise. In order to avoid this, a 
fade-in/out process is desirably carried out at around a 
signal rise and fall to obtain a smooth rise/fall charac- 
teristic. The fade-in/out process can be easily realized 
by a known method using a fade-in/out function. In Fig. 

55 1 7, W1 - W4 are regions that the fade-in/out process is 
to be carried out. 

[0087] Fig. 1 8 is a flowchart in the case the silent sec- 
tions are cut away to record the data. Data is read se- 
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quentially from the top (S1 71 ), and whether the level of 
the read data exceeds a constant value or not (S1 72) is 
determined. If the constant value is not exceeded (S1 72 
NO), S1 71 is returned to subsequently read the data. If 
the constant value is exceeded (S1 72 YES), the forego- 
ing fade-in/out process is carried out at around a rise in 
the data and the data after the process is written to the 
memory (S173). The fade-in/out process herein is a 
fade-in/out process in W1 of Fig. 17, which is a fade-in 
process that the signal rises moderately. 
[0088] Next, data is read in again (S1 74), and whether 
the level of the read data exceeds a constant value or 
not is determined (S175). If the constant value is ex- 
ceeded (S175 YES), the data is written to the memory 
(S1 76) and S1 74 is returned to read the next data. If the 
constant value is not exceeded (S175 NO), it is deter- 
mined whether the section has continued a constant 
section or not (S1 77). If not continued a constant section 
(S1 77 NO), the data is written to the memory (S1 76) and 
S174 is returned to read the next data. If a section a 
constant level is not exceeded continues a constantsec- 
tion (S177 YES), the section is considered as a silent 
section so that a fade-in/out process is made in the re- 
gion W2 in Fig. 1 7 and the data after the process is writ- 
ten to the memory (S1 78). The fade-in/out process here- 
in is a fade-out process that the signal falls moderately. 
Incidentally, in S178 is carried out a process to erase 
the unnecessary data in the silent sections among the 
data written in S176. 

[0089] Next, whether data reading is ended or not is 
determined (S1 79). If not ended (S1 79 NO), S1 71 is re- 
turned to read in the next data. From now on, the similar 
processes to the above are passed to carry out a fade- 
in/out process in W3, W4 of Fig. 1 7. If the data reading 
is ended (S179 YES), the operation is ended. 
[0090] In the above embodiment, audio, text and im- 
ages are taken as information to be buried in an SMF 
extended format. However, the burying information may 
be anything, e.g. a computer program is satisfactory. In 
this case, if a computer program is provided to be repro- 
duced following MIDI data, how to use is feasible such 
that a music according to the MIDI is first played and, 
when this is ended, the program is automatically run. 
[0091 ] Also, although the above embodiment showed 
the example that data is received from a file 1 a of a serv- 
er on a network through a communication line, it is sat- 
isfactory to create SMF-type data on a personal compu- 
ter and store it in a file 1 b on the hard disk so as to down- 
load the data therefrom. 

[0092] Fig. 19 shows another example of a data re- 
producing apparatus according to the invention. 1a, 1b 
are files on which data is recorded. 1a is, for example, 
a file in a server on the Internet, and 1 b is, for example, 
a file on a hard disk within the apparatus. 
[0093] 2 is a CPU for controlling the overall of the data 
reproducing apparatus, which is configured including a 
data receiving section 3 and a data assorting section 4. 
Although besides these the CPU 2 includes the blocks 



having various functions, they in the invention are not 
related and omittedly shown. The data receiving section 
3 accesses the files 1a, 1 b to receive the data stored in 
these. The data of the file 1 a is received through a wire 
5 or wirelessly. The format of the data to be received is 
the same as Fig. 2 to Fig. 8. These of received data are 
temporarily stored in a buffer 3a. The data assorting sec- 
tion 4 assorts the data received by the data receiving 
section 3 on a kind-by-kind basis and stores them in the 
10 buffers 7-10 constituting a storage section 5. 

[0094] 6 is a data reproducing section configured with 
a MIDI reproducing section 11 for processing the data 
concerning MIDI, an audio reproducing section 12 for 
processing the data concerning audio, a text reproduc- 
es ing section 13 for processing the data concerning text, 
and an image reproducing section 1 4 for processing the 
data concerning images. Incidentally, although not 
shown, the MIDI reproducing section 11 has a sound- 
source ROM 1 1 a of Fig. 1 . The image reproducing sec- 
20 tion 1 4 has a function to reproduce still images and mo- 
tion images. 

[0095] 1 5 is a mixer for mixing the outputs of the M I Dl 
reproducing section 11 and audio reproducing section 
12, and 16 is a mixer for mixing the output of the text 

25 reproducing section 13 and image reproducing section 
14. Although herein not shown, the mixer 15 has a 
sound-effect section 1 5a of fig. 1 while the mixer 1 6 has 
a visual-effect section 1 6a of Fig. 1 . 1 7 is an output buff- 
er to temporarily store the output of the mixer 15, and 

30 1 8 is an output buffer to temporarily store the output of 
the mixer 1 6. 1 9 is aspeaker as asound generating sec- 
tion to output sound on the basis of the data of the output 
buffer 1 7, and 20 is a display for displaying visual infor- 
mation, such as characters and illustrations, on the ba- 

35 sis of the data of the output buffer 1 8. 21 is a timing con- 
trol section for generating a system clock providing a 
reference time to control the timing of each section, and 
22 is an external storage device to be externally at- 
tached to the data reproducing apparatus. 

40 [0096] The storage section 5, the data reproducing 
section 6, the mixers 15, 16, the output buffers 17, 18 
and the timing control section 21 are configured by a 
DSP (Digital Signal Processor). The above sections can 
be configured by a LSI, in place of the DSP. 

45 [0097] As apparent from a comparison between Fig. 
1 9 and Fig. 1 , the data reproducing apparatus of Fig. 1 9 
has a storage section 5 comprising buffers 7-10 be- 
tween the data assorting section 4 and the data repro- 
ducing section 6, and also has a timing control section 

50 21. Furthermore, an external storage device 22 is also 
added. 

[0098] Fig. 20 is a flowchart showing the overall op- 
eration of the data reproducing apparatus of Fig. 19. 
First, the data receiving section 3 receives data from the 
55 file 1 a or file 1 b (S1 81 ). This received data is stored to 
the buffer 3a. Next, the CPU 2 carries out a time oper- 
ation required for data assortment by the data assorting 
section 4 on the basis of the system clockfrom the timing 



10 



19 



EP 1 172 796 A1 



20 



control section 21 and delta time of each of data re- 
ceived by the data receiving section 3 (S1 82). The detail 
of S182 will be hereinafter referred to. The data assort- 
ing section 4 assorts the data to be processed on a kind- 
by-kind basis depending upon a result of time operation, 
and stores them to the corresponding buffers 7-10 
(S183). The detail of S183 will also be hereinafter re- 
ferred to. 

[0099] The data stored in the buffer 7 - 1 0 is read out 
by the data reproducing section 11-14 corresponding 
to each buffer so that the event recorded in the data is 
executed in each data reproducing section 11-14 there- 
by reproducing the data (S1 84). The detail of S1 84 will 
be hereinafter referred to. Among the reproduced data, 
the data of MIDI and audio is mixed in the mixer 15 while 
the data of text and images is mixed in the mixer 16 
(S185). These of mixed data are respectively stored in 
the output buffers 17,18 and thereafter outputted to the 
speaker 19 and the display 20 (S186). 
[0100] Fig. 21 is a figure explaining a principle of time 
operation in S182. In the figure, t is a time axis and an 
event 0 to event 4 show the timing of reproducing an 
event included in a received data row (it should be noted 
that this reproducing timing represents a timing in the 
case that the received data is assumed to be repro- 
duced according to a delta time thereof, instead of rep- 
resenting an actually reproduced timing on the time axis 
t) . For example, the event 0 is an image event, the event 
1 is a MIDI event, the event 2 is an audio event, the 
event 3 is atext event, and the event 4 is an image event. 
AT1 to AT4 are a delta time, wherein AT1 is a delta time 
of the event 1 , AT2 is a delta time of the event 2, AT3 is 
a delta time of the event 3, and AT4 is a delta time of 
the event 4. As in the foregoing, the delta time is a time 
of from a time point of executing the immediately pre- 
ceding event to an execution of the current event. For 
example, the event 2 is executed upon elapsing AT2 
from the time point of executing the event 1, and the 
event 3 is executed upon elapsing AT3 from the time 
point of executing the event 2. t1 represents a time that 
the last-time data has been processed, and t2 a current 
time. The difference t2 - 11 corresponds to one frame as 
a unit section. This one-frame section has a time width, 
for example, of 15 ms. The first and last timings in one 
frame are determined by the system clock from the tim- 
ing control section 21 (see Fig. 1 9). Q is a data process- 
ing section, which is defined as a difference between a 
current time t2 and an execution time to of the last event 
in a frame preceded by one (event 0). 
[0101] Fig. 22 is a flowchart showing a procedure of 
data assortment by the data assorting section 4. Here- 
under, explanation is made on the procedure of assort- 
ing data, with reference to Fig. 21 and Fig. 22. In the 
timing t2 of Fig. 21 , if there is a clock interrupt from the 
timing control section 21 to the CPU 2, the system be- 
comes a WAKE state (S1 91 ) so that the CPU 2 operates 
a time width of a process section Q (S192). This Q, as 
in the foregoing, is calculated as: 



Q = t2 - tO, 

which represents a time width for processing the current 

5 data. Next, the CPU 2 sequentially reads a delta time 
AT of the received data (S1 93) to determine whether the 
time width of the process section Q is equal to or greater 
than AT or not (S1 94). If Q > AT (S1 94 YES), data kind 
determinations are next made in order (S195, S198, 

10 S200, S202), thereby the data is assorted and stored to 
the buffers 7-10 provided corresponding to the respec- 
tive ones of data (S1 96, S1 99, S201 , S203). Thereafter, 
Q = Q - AT is operated to update the value of Q (S1 97). 
[0102] In the example of Fig. 21, because the event 

15 0 has been already processed in the last time, determi- 
nation is made in order from the event 1. Concerning 
the delta time AT1 of the event 1 , because of Q > AT1 , 
the determination in S1 94 is YES and next whether the 
data is MIDI or not is determined (S195). In Fig. 21, if 

20 the event 1 is a MIDI event (S1 95 YES), the data is for- 
warded to and temporarily stored in the buffer 7 (S1 96). 
If the event 1 is not a MIDI event (S195 NO), whether 
an audio event or not is determined (S1 98). If the event 
1 is an audio event (S198 YES), the data is forwarded 

25 to and temporarily stored in the buffer 8 (S199). If the 
event 1 is not an audio event (S1 98 NO), whether a text 
event or not is determined (S200). If the event 1 is a text 
event (S200 YES), the data is forwarded to and tempo- 
rarily stored in the buffer 9 (S201). If the event 1 is not 

30 a text event (S200 NO), whether an image event or not 
is determined (S202). If the event 1 is an image event 
(S202 YES), the data is forwarded to and temporarily 
stored in the buffer 10 (S203). If the event 1 is not an 
image event (S202 NO), another process is carried out. 

35 [0103] In this manner, the data of the event 1 is as- 
sorted to any of the buffers 7-10, and thereafter Q = Q 
-AT1 is operated (S197). Returning to S1 93, adelta time 
AT2 of the next event 2 is read to determine Q > AT2 
(S194). Although the value of Q at this time is Q = Q - 

40 AT1 , because Q - AT1 > AT2 in Fig. 21 , the determina- 
tion in S194 is YES. Similarly to the above, data kind 
determinations of the event 2 are made for assortment 
into the corresponding buffers. 

[0104] Thereafter, Q = Q - AT2 is operated (S197). 

45 Returning to S1 93, a delta time AT3 of the next event 3 
is read to determine Q > AT3 (S1 94). Although the value 
of Q at this time is Q = Q - AT1 - AT2, because of Q - 
AT1 - AT2 > AT3 in Fig. 21, the determination in S194 
is YES. Similarly to the above, data kind determinations 

50 of the event 3 are made for assortment to the corre- 
sponding buffers. 

[0105] Thereafter, Q = Q - AT3 is operated (S197). 
Returning to S1 93, a delta time AT4 of the next event 4 
is read (Although in Fig. 21 the event 4 is shown later 
55 than t2, the data of the event 4 at a time point t2 has 
already entered the buffer 3a and is possible to read out) 
to determine Q > AT4 (S194). Although the value of Q 
at this time is Q = Q - AT1 - AT2 - AT3, because of Q - 
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AT1 - AT2 - AT3 < AT 4 in Fig. 21, the determination in 
S194 is NO. The CPU 2 does not carries out data 
processing of the event 4 but enters a SLEEP state to 
stand by until the process in the next frame (S204). 
Then, if there is a clock interrupt in the first timing of the 
next frame from the timing control section 21 , a WAKE 
state is entered (S191), whereby the data of the event 
4 and the subsequent are processed similarly to the 
foregoing process. 

[0106] In the flowchart of Fig. 22, S192 - S194 and 
S1 97 are the details of S1 82 of Fig. 20, and S1 95, S1 96 
and S1 98 - S203 are the details of S1 83 of Fig. 20. 
[0107] Next, explanation is made on the detail of each 
data reproducing section 11 - 14, i.e. the detail of S184 
of Fig. 20. Fig. 23 is a flowchart showing a process pro- 
cedure in each data reproducing section, wherein (a) 
represents a process procedure in the MIDI reproducing 
section 11 . In the MIDI reproducing section 11 , when the 
dataof one-frame section assorted by the data assorting 
section 4 is stored to the buffer 7, this data is read in the 
next one-frame section (S211). Then, the content of a 
MIDI event recorded in the read data (see Fig. 3, Fig. 4) 
is decrypted to create a synthesizer sound by a software 
synthesizer (S212). The output of the synthesizer is 
temporarily stored in a not-shown buffer present within 
the MIDI reproducing section 11, and outputted to the 
mixer 15 from this buffer (S213). 
[0108] Fig. 23(b) shows a process procedure in the 
audio reproducing section 12. In the audio reproducing 
section 1 2, when the data of one-frame section assorted 
by the data assorting section 4 is stored to the buffer 8, 
this data is read in the next one-frame section (S311). 
Then, the audio data recorded in an event of the read 
data (see Fig. 5(b), Fig. 7(b)) is decoded to reproduce 
an audio (S312). The reproduced data is temporarily 
stored in a not-shown buffer present within the audio re- 
producing section 1 2, and outputted to the mixer 1 5 from 
this buffer (S31 3). 

[0109] Fig. 23(c) shows a process procedure in the 
text reproducing section 13. In the text reproducing sec- 
tion 1 3, when the data of one-frame section assorted by 
the data assorting section 4 is stored to the buffer 9, this 
data is read in the next one-frame section (S41 1 ). Then, 
the text data recorded in an event of the read data (see 
Fig. 5(c), Fig. 7(c)) is decoded to reproduce a text 

(5412) . The reproduced data is temporarily stored in a 
not-shown buffer present within the 'text reproducing 
section 1 3, and outputted to the mixer 1 6 from this buffer 

(5413) . 

[0110] Fig. 23(d) shows a process procedure in the 
image reproducing section 1 4. In the image reproducing 
section 1 4, when the data of one-frame section assorted 
by the data assorting section 4 is stored to the buffer 1 0, 
this data is read in the next one-frame section (S511). 
Then, the image data recorded in an event of the read 
data (see Fig. 5(d), Fig. 7(d)) is decoded to reproduce 
an image (S512). The reproduced data is temporarily 
stored in a not-shown buffer present within the image 



reproducing section 14, and outputted to the mixer 16 
from this buffer (S513). 

[0111] Each process of Figs. 23(a)-(d) stated in the 
above is carried out according to a sequence deter- 

5 mined by a program, herein assumably carried out in 
the sequence of (a) to (d). Namely, the MIDI process of 
(a) is first made. If this is completed, the audio process 
of (b) is entered. If the audio process is completed, the 
text process of (c) is entered. If the text process is com- 

10 pleted, the image process of (d) is made. Incidentally, 
the reason of carrying out the processes in aseries fash- 
ion is that the DSP constituting the storage section 5, 
the data reproducing section 6 and the like is one in the 
number. Where a DSP is provided in each reproducing 

15 section, the process can be made in a parallel fashion. 
[0112] The MIDI reproduced data outputted to the 
mixer 1 5 in S21 3 and the audio reproduced data output- 
ted to the mixer 15 in S313 are mixed together in the 
mixer 1 5 and stored to the output buffer 1 7, thereby be- 

20 ing outputted as a sound from the speaker 1 9. Also, the 
text reproduced data outputted to the mixer 16 in S413 
and the image reproduced data outputted to the mixer 

1 6 in S51 3 are mixed together in the mixer 1 6 and stored 
to the output buffer 1 8, thereby being displayed as avis- 
os ible information on the display 20. The output buffer 1 7 

and the speaker 19 constitute a first output section, 
while the output buffer 18 and the display 20 constitute 
a second output section. Incidentally, the output buffer 

1 7 has a function to count the number of the data to be 
30 outputted to the speaker 1 9. On the basis of this count 

value, the output buffer 17 supplies a control signal to 
the timing control section 21 . The timing control section 
21 supplies a timing signal (system clock) to the CPU 2 
on the basis of the control signal. Namely, the time re- 

35 quired in outputting the data of one in the number from 
the output buffer 17 is determined by a sampling fre- 
quency. If this time is given x, the time required in out- 
putting the data of N in the number is N x x. Accordingly, 
the timing can be determined by a value of N. Mean- 

40 while, the timing control section 21 supplies the timing 
signal also to the output buffer 1 8 according to the con- 
trol signal, to control the timing of the data to be output- 
ted from the output buffer 18. 

[01 13] Fig. 24 is a figure showing in an overall fashion 
45 the foregoing operation of from data assortment to re- 
producing, wherein (a) represents a relationship be- 
tween an amount of data to be processed in each repro- 
ducing section and a frame section and (b) represents 
a relationship between a process time in each reproduc- 
50 ing section and a frame section. F1 - F3 are one-frame 
sections, wherein the time width of each frame section 
is set, for example, at 1 5 ms. Namely, the data assorting 
section 4 is interrupted by a clock at a time interval of 
15 ms from the timing control section 21 . t represents a 
55 time axis, M a reproducing timing of a MIDI event, A a 
reproducing timing of an audio event, T a reproducing 
timing of a text event and P a reproduce timing of an 
image event. Incidentally, these reproducing timing 
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show the timings on the assumption that the received 
data be reproduced according to a delta time similarly 
to Fig. 21 , instead of showing the timings that actual re- 
producing is made on the time axis t. 
[0114] As was explained in Fig. 21, the data to be 
processed in a section F1 is assorted and stored to the 
buffers 7 - 1 0 in the last timing of the same section. Each 
reproducing section 11-14 reads data from the buffer 
in the next one-frame section F2 to carry out a repro- 
ducing process. In this case, the amount of data to be 
transferred from each buffer to each reproducing sec- 
tion is data in an amount that each reproducing section 
can process in the one-frame section. As shown in Fig. 
24(a), each reproducing section is made to process all 
the data within the next one-frame section F2. 
[0115] The time chart of this process is Fig. 24(b), 
wherein the length of a white arrow represents a process 
time. The process time is different depending on each 
frame. As in the foregoing, the data stored in the buffer, 
in the next one frame section F2, is sequentially read 
out in a predetermined order by each reproducing sec- 
tion 11 - 1 4, so that in each reproducing section an event 
recorded in the data is executed thereby reproducing 
the data. In Fig. 24(b), M (MIDI), A (audio) and P (image) 
are reproduce-processed in this order. The reproduced 
M and A are processed in a mixer 1 (mixer 1 5 in Fig. 1 9) 
while the reproduced P is processed in a mixer 2 (mixer 
16 in Fig. 19). In this manner, the data assorted in the 
section F1 is all completed in process within the section 
F2. The remaining time is a standby time before starting 
a process in a section F3. This is shown by SLEEP in 
the figure. The output from the mixer 1 is stored in an 
output buffer 1 (output buffer 1 7 in Fig. 1 9) and thereaf- 
ter outputted as a sound in the next frame section F3. 
Meanwhile, the output from the mixer 2 is stored to an 
output buffer 2 (output buffer 1 8 in Fig. 1 9) and thereaf- 
ter outputted as visible information in the frame section 
F3. 

[0116] Similarly, in the section F2, the data A, M, T is 
assorted to the buffer. These of data are read out in the 
order of M, A and T in the section F3, reproduce-proc- 
essed in the same procedure as the above in each re- 
producing section and outputted in the next section F4 
(not shown in Fig. 24). 

[0117] In the above manner, in the data reproducing 
apparatus of Fig. 1 9, the received data is assorted on a 
frame-by-frame basis and stored to the buffer, and read 
out from the buffer in the next frame to reproduce the 
data, and outputted as a sound or visible information in 
the further next frame. Accordingly, reproducing can be 
made while taking time synchronization of data on a 
frame-unit basis. 

[01 1 8] Meanwhile, the data assorting section 4 is de- 
voted itself to the operation of assorting the received da- 
ta to the buffers 7-10, and each reproducing section 1 1 
- 14 is devoted itself to reading out and reproducing the 
data stored in the buffer. Accordingly, it is possible to 
make the data received by the data receiving section 3 



into a pipeline form and process it at high speed. 
[0119] Incidentally, in reproducing data, the timing of 
reproducing should be primarily controlled according to 
delta time. However, in the apparatus of Fig. 19, after 

5 data is assorted to the buffers 7 - 1 0 by the data assort- 
ing section 4, the data is separated and hence the indi- 
vidual delta time has substantially insignificance in de- 
termining a reproducing timing. However, because one 
frame section as above is an extreme short time of 15 

10 ms, it is satisfactory to consider that the data reproduced 
in this duration have been simultaneously reproduced 
regardless of the reproducing timing of each of data. Ac- 
tually, it is empirically ascertained that the deviation in 
data reproducing timing within a section of nearly 1 5 ms 

15 cannot be distinguished by a usual human sense. Ac- 
cordingly, upon assorting data if deciding data that is to 
be processed within one frame section on the basis of 
delta time, there is no problem even where, within one 
frame section, the reproducing timing of the data is de- 

20 viated from the reproducing timing according to delta 
time. 

[0120] Furthermore, it is satisfactory that the order of 
reproducing different kinds of data be changed within 
the same frame section. For example, although each re- 

25 producing section reads data from the buffer according 
to the order of M, A and P of the received data in the F1 
section of Fig. 24(b), in the section F2 the order the re- 
producing sections reads data out of the buffers is M, A 
and T, i.e. A and M exchanged despite the order of re- 

30 ceived data is A, M and T. This is because, as described 
before, the process order in each reproducing section 
is fixed as M, A, T and P by a program. However, even 
if the process order is changed in this manner, if each 
reproducing section carries out data processing within 

35 1 5 ms, there is no problem because the reproducing tim- 
ing of data is not known by a human sense as described 
before. 

[01 21 ] Meanwhile, although in Fig. 24 the data assort- 
ed in the one frame section is all processed within the 

40 next one frame section, this is not essentially required. 
Namely, if the output buffers 1 7, 1 8 have a size exceed- 
ing the processing amount in one frame section, even 
where there is data having not been processed within 
the one frame, the earlier-processed data is left in the 

45 output buffers 17,18 and hence the data can be output- 
ted without suspension. 

[0122] Fig. 25 is a figure explaining the operation of 
the data receiving section 3 where a stream scheme for 
carrying out reproducing while downloading data in the 

50 data reproducing apparatus of Fig. 1 or Fig. 19. Herein, 
a buffer 3a is configured by three buffers of buffer A, 
buffer B and buffer C. 3b is registers A, B, C correspond- 
ing to the buffers A, B, C. The data to be received is 
shown as stream data S. The stream data S has a head- 

55 er H recorded at its top. Following this, MIDI, audio, text 
and images of data are mixedly recorded as packets P1 , 
P2, P3, ... Pm. The total data amount of the stream data 
S is given K. 
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[0123] Hereunder, explanation is made on the receiv- 
ing operation with an example in the case of reproducing 
music. When the data receiving section 3 starts to re- 
ceive stream data S from the file 1 a due to access to the 
server, the data A1 in an amount corresponding to a 
buffer A size (capacity) is first stored from the top of the 
stream data S to the buffer A. This makes the buffer A 
full in state, to set the register A with a flag representa- 
tive of a full state of the buffer A. Subsequently, the data 
B1 in an amount corresponding to a size of the buffer B 
is stored to the buffer B. This makes also the buffer B 
full in state, to set the register B with a flag representa- 
tive of a full state of the buffer B. 
[0124] At a time point that the buffer B become full, 
the data assorting section 4 start to assort data thereby 
transferring the data A1 stored in the buffer A and the 
data B1 stored in the buffer B to the buffer 7 - 10 on a 
kind-by-kind basis of data. The transferred data is re- 
produced by each reproducing section 11 - 14, starting 
music to play. On the other hand, the data C1 is stored 
to the buffer C in an amount corresponding to the size 
thereof. This makes the buffer C full in state, to set the 
register C with a flag representative of a full state of the 
buffer C. 

[0125] During storage of data C1 in the buffer C, if the 
data A1 of the buffer A is consumed to make the buffer 
A empty, a flag of the register A is reset. The data re- 
ceiving section 3 acquires the next data A2 and stores 
it to the buffer A. This makes the buffer A again in a full 
state, to set a flag in the register A. Meanwhile, if the 
data B1 of the buffer B is consumed to make the buffer 
B empty, a flag of the register B is reset. The data re- 
ceiving section 3 acquires the next data B2 (not shown 
in Fig. 25) and stores it to the buffer B. This makes the 
buffer B again in a full state, to set a flag in the register 
B. By repeating the above operation, the reproducing of 
stream data S proceeds. Fig. 26 is a figure showing a 
flow of data in this case. 

[0126] In the above stream scheme, reproducing is 
possible to start from a time point of receiving the data 
A1 . However, where the transfer capacity of the data to 
be fetched to the buffer is insufficient, after a reproduce 
start the data supply to the buffer does not catch up with 
consumption thus causing a phenomenon of disconti- 
nuity in sound. Accordingly, in order to avoid this, there 
is a need of caching data to the buffer to start reproduc- 
ing at a time point a certain degree of data is saved. This 
will be explained on an example of Fig. 27. 
[0127] Assuming in Fig. 27 that the buffers A, B, C are 
respectively in size of 50 K-bits and the time required in 
fetching data to the buffer is 5 seconds, the data transfer 
capacity per second is given 50/5 = 10 K-bps. Mean- 
while, assuming that a music-playing time of is 10 sec- 
onds and the total data amount is 200 K-bits, the amount 
of data consumed by playing the music is 200/10 = 20 
K-bps. Consequently, the reproducing if started at a time 
point to of receiving data makes the amount of con- 
sumed data in excess of the amount of data fetched to 



the buffer, resulting in insufficient data in the buffer and 
discontinuity in sound. 

[0128] This problem is to be solved as follows. Name- 
ly, 50 K-bits of data A1 is stored to the buffer A in 5 sec- 

5 onds from a time point to of receiving data, and 50 K- 
bitsofdataBI is stored to the buffer B in the subsequent 
5 seconds. The data of totally 100 K-bits is cached for 
1 0 seconds. Then, reproducing is started at a time point 
t1 that 10 seconds have passed from a data-reception 

10 time point to. By doing this, even if the data transfer ca- 
pacity after a reproduce start is smaller than the amount 
of data consumption, the buffers A, B are already saved 
with 1 00 k-bits of data. Also, because the remaining 1 00 
k-bits of data (the total of C1 and A2) can be fetched to 

15 the buffers C, A for 10 seconds of from the music-play 
start time point t1 to the music-play end time point t2, 
there encounters no data exhaustion and hence the mu- 
sic can be reproduced continuously to the end. 
[0129] Contrary to this, where the amount of data to 

20 be taken to the buffer exceeds the amount of consumed 
data, the data cache as above is not required. However, 
there is a need, at a time the buffer becomes a full state, 
of providing an instruction to the server from the data 
receiving section 3 not to transmit further data. In this 

25 case, when the data of the buffer is consumed to cause 
an empty in the buffer, the data receiving section 3 will 
acquire data from the server. 

[01 30] The foregoing is generalized to describe as the 
following. Assuming that the buffer size is U and the time 

30 required in fetching data to the buffer is t, the data trans- 
fer capacity per unit time is given J = U/t. Meanwhile, 
the total data amount is K and the reproducing time is 
T, the data consumption amount E per unit time is given 
E = K/T. In Fig. 25, the total data amount K and the mu- 

35 sic-play time T is recorded in the header H, so that the 
data receiving section 3 calculates a data consumption 
amount E by reading the header H. Also, at a time that 
data A1 is fetched to the buffer A, the data transfer ca- 
pacity J is calculated. As a result, if J < E, it is determined 

40 that data cache is required thereby caching a required 
amount of data. In this case, if data is cached, with a 
data cache amount of C, to meet the condition of 

45 K < C + J • T, 

then it is possible to reproduce the data without discon- 
tinuity. In order to cache data, the data receiving section 
3 acquires data B1 from the server and store it to the 

50 buffer B. If the above condition is fulfilled at this time 
point, the data receiving section 3 forwards a ready sig- 
nal to the data assorting section 4. Receiving this, the 
data assorting section 4 starts to assort the data to the 
buffers A, B. The operation from then on is as per the 

55 foregoing. 

[0131] On the other hand, if J > E, data cache is not 
required. Consequently, the data assorting section 4 
starts to assort the data at a time of receiving the data 
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A1 . However, because the buffer immediately becomes 
full in state after a start of reproducing, the data receiv- 
ing section 3 requests the server to stop the transmis- 
sion of data at a time the buffer becomes a full state. 
Then, if the data is consumed to cause a free space in 
the buffer, the data receiving section 3 again requests 
the server to transmit data. Namely, the data receiving 
section 3 intermittently acquires data from the server. 
[0132] In the above manner, the data receiving sec- 
tion 3 monitors the data transfer capacity J. If J < E, data 
is cached in a required amount and thereafter reproduc- 
ing is started. If J > E, data cache is not made to carry 
out reproducing while intermittently receiving data. This 
makes it possible to stably reproduced data, regardless 
of variation in transmission line capacity. Incidentally, in 
the case of J = E, data cache is not necessary so that 
data is continuously received from the server. 
[0133] Herein, if the transmission line capacity is sud- 
denly decreased due to a certain cause, the data cache 
to the buffer is possibly insufficient resulting in an empty 
state in all the buffers A, B, C. In this case, a mute signal 
is forwarded from the data assorting section 4 to the 
MIDI reproducing section 1 1 and audio reproducing sec- 
tion 12 to prohibit noise from being outputted, thereby 
eliminating an uncomfortable feeling to the user. Also, it 
is preferred to forward a front-end hold signal from the 
data assorting section 4 to the text reproducing section 

13 and image reproducing section 14 thereby maintain- 
ing an on-screen display immediately before. Also, in 
place of these, where no data comes from the data as- 
sorting section 4 despite each reproducing section 11 - 

14 has not receive a signal representative of data end, 
it is also possible to adopt a method that a mute or front- 
end hold process is automatically made in each repro- 
ducing section 11 - 14 to resume reproducing if data 
comes. 

[0134] In the above explanation, although independ- 
ent three buffers A, B, C were provided as the buffer 3a, 
this is mere one example and the number of buffers can 
be arbitrarily selected. Also, a ring buffer may be used 
in place of the independent buffers. 
[0135] Next, explanation is made on application ex- 
amples of the invention. The data reproducing appara- 
tus of Fig. 1 or Fig. 1 9 can be mounted on an information 
terminal having a function of a telephone. Due to this, 
realized is a cellular phone that can download a variety 
of information, such as sound, text and images, and re- 
produce them to produce sound through the speaker or 
display text or images on the display. For example, the 
CMs (commercials) or music and images, such as of ka- 
raoke, offered by the Internet can be viewed on the cel- 
lular phone. An example of such a cellular phone is 
shown in Fig. 37. 

[0136] In Fig. 37, 50 is a cellular phone as an infor- 
mation terminal and 51 is a main body of the phone, 
wherein the main body 51 is provided with an antenna 
52, a display 53, various keys such as a numeric key 
54, a speaker 55 and a microphone 56. This cellular 



phone 50 communicates with a base station 73, to 
download the data stored in a server 72 through the 
base station 73. 

[0137] The antenna 52 is to transmit and receive sig- 
5 nals to and from the base station 73. The display 53 is 
structured by a color liquid crystal display and the like, 
to display telephone numbers, images and so on. From 
the speaker 55 as a sound generating section is to be 
heard a voice of the opposite of communication or a mel- 
10 ody. The microphone 56 is to input speech during com- 
munication or in preparing an in-absence guide mes- 
sage. 

[01 38] 54 is a numeric key comprising numerals 0 - 9, 
which is used to input a telephone number or abbrevi- 

15 ated number. 57 is a power key for turning on/off power 
to the phone, 58 is a talk key to be operated upon start- 
ing a talk, and 59 is a scroll key for scrolling the content 
displayed on the display 53. 60 is a function key for 
achieving various functions by the combined operation 

20 with another key, 61 is an invoking key for invoking a 
content registered and displaying it on the display 53, 
and 62 is a register key to be operated in registering 
abbreviated dial numbers, etc. 63 is a clear key for eras- 
ing a display content or the like, and 64 is an execute 

25 key to be operated in executing a predetermined oper- 
ation. 65 is a new-piece display key for displaying a list 
of new pieces in downloading the music data from the 
server 72, 66 is an in-absence recording key to be op- 
erated in preparing an in-absence guide message, 67 

30 is a karaoke key to be operated in playing karaoke, 68 
is a music-play start key for starting a music play, 69 is 
a music-play end key for ending a music play. 
[0139] Meanwhile, 70 is a small-sized information 
storage medium in the form of a card, a stick or the like 

35 which is removably attached in a slot (not shown) pro- 
vided in the phone main body 51 . This information stor- 
age medium 70 incorporates therein a flash memory 71 
as a memory device. A variety of data downloaded is 
stored in this memory 71 . 

40 [0140] In the above structure, the display 53 corre- 
sponds to the display 20 of Fig. 1 or Fig. 19, to display 
a text or image herein. For example, in the case of CM, 
text, illustrations, pictures, motion pictures or the like are 
displayed. In the case of karaoke, titles, words, back- 

45 ground images are displayed. Meanwhile, the speaker 
55 corresponds to the speaker 1 9 of Fig. 1 or Fig. 1 9, to 
output sound of MIDI or audio from here. For example, 
in the case of CM, CM songs or item guide messages 
are sounded. In the case of karaoke, accompaniment 

50 melodies or back choruses are sounded. In this manner, 
by mounting the data reproducing apparatus of Fig. 1 or 
Fig. 19 on the cellular phone 50, the cellular phone 50 
can be utilized, for example, as a karaoke apparatus. 
[0141] Meanwhile, MIDI data only can be downloaded 

55 from the server 72 onto the cellular phone 50. In this 
case, if a melody created by the MIDI is outputted as an 
incoming tone from the speaker 55, the incoming tone 
will be an especially realistic, refined music. Also, if dif- 
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ferent music of MIDI data is stored correspondingly to 
incoming tones to an internal memory (notshown) of the 
cellular phone 50 to notify with different melodies de- 
pending upon an incoming tone, it is possible to easily 
distinguish from whom the call is. Meanwhile, an incom- 
ing-call-notifying vibrator (not shown) built in the cellular 
phone 50 may be vibrated on the basis of MIDI data, e. 
g. the vibrator is vibrated in the same rhythm as a drum. 
Furthermore, such a use way is possible as to add BGM 
(BackGround Music) due to MIDI to in-absence guide 
messages. 

[0142] The information storage medium 70 corre- 
sponds to the external storage device 22 of Fig. 19, 
which can store and save music data or image data in 
the flash memory 71 . For example, where downloading 
CD (Compact Disk) music data, as was shown in Fig. 
38 the information storage medium 70 itself can be 
made as a CD by recording CD-jacket picture data due 
to images in addition to music data due to MIDI or audio, 
data of music words and commentaries and the like due 
to text. This is true for the case of MD (Mini Disk). 
[0143] In the cellular phone 50 mounted with the data 
reproducing apparatus as above, for example where 
there is an incoming call during viewing CM, it is desired 
to preferentially output an incoming tone. Fig. 28 shows 
a configuration for realizing this. The apparatus of Fig. 
28 is also to be mounted on a cellular phone 50, wherein 
the identical parts to Fig. 19 are attached with the iden- 
tical reference numerals. In Fig. 28, the difference from 
Fig. 1 9 is in that an incoming-signal buffer 23 is provided 
and a switch section 24 is provided between the buffer 
7 and the MIDI reproducing section 11. 
[0144] Fig. 29 is a time chart showing the operation 
of the data reproducing apparatus of Fig. 28. It is as- 
sumed that, first, a CM music is sounded as in (c) from 
the speaker 1 9 while a CM image is displayed as in (d) 
on the display 20. Now, provided that an incoming-call 
signal as in (a) is inputted as an interrupt signal to the 
data receiving section 3, the data receiving section 3 
stores the data of the incoming-call signal to the buffer 
23 and switches the switching section 24 from a buffer 
7 to buffer 23 side. This inputs the data of the buffer 23 
in place of the data of the buffer 7 to the MIDI reproduc- 
ing section 11. The MIDI reproducing section 11 reads 
the data of the buffer 23 to create an incoming tone by 
the software synthesizer and outputs it to the speaker 
1 9 through the mixer 1 5 and output buffer 1 7. As a result, 
a MIDI incoming tone in place of the CM music is out- 
putted as in (b) from the speaker 19. Then, when the 
incoming of signal is ended to cease the incoming tone, 
a CM music is again sounded as in (c) from the speaker 
1 9. Incidentally, the CM image is continuously displayed 
on the display 20 as in (d) regardless of the presence 
or absence of an incoming tone. In this manner, accord- 
ing to the data reproducing apparatus of Fig. 28, when 
there is an incoming signal, a incoming tone is prefer- 
entially outputted thus enabling the viewer to positively 
know the signal incoming. Also, in creating an incoming 



tone, because the software synthesizer of the MIDI re- 
producing section 11 can be commonly used, the proc- 
ess is simplified. 

[0145] The data reproducing apparatus of the inven- 

5 tion can be mounted, for example, on an information ter- 
minal having a function of game machine besides on an 
information terminal having a function of telephone. The 
game machine may be a game exclusive machine or an 
apparatus possessing both game and other functions. 

10 For example, satisfactory is the cellular phone 50 shown 
in Fig. 37 installed with software of a game. 
[0146] In a game machine alike this, although, usual- 
ly, music sounds in the background during progress of 
a game, an interesting game development is available 

15 if an effect sound due to MIDI is produced over a back 
music to the on-screen situation. Fig. 30 is a configura- 
tion for realizing this, wherein the identical parts to Fig. 
9 is attached with the identical reference numerals. In 
Fig. 30, the difference from Fig. 19 lies in that an effect- 

20 sound signal buffer 25 is provided and a mixer 26 is pro- 
vided between the buffer 7 and the MIDI reproducing 
section 11. 

[0147] Fig. 31 is a time chart showing the operation 
of the apparatus of Fig. 30. It is assumed that, first, a 

25 back music sounds as in (c) from the speaker 1 9 and a 
game image is displayed as in (d) on the display 20. 
Now, provided that an effect-sound signal as in (a) is 
inputted as an interrupt signal to the data receiving sec- 
tion 3 by operating a particular button of the game ma- 

30 chine, the data receiving section 3 stores the data of the 
effect sound signal to the buffer 25. The effect-sound 
data of the buffer 25 is mixed with the data of the buffer 
7, in the mixer 26. The MIDI reproducing section 11 
reads the data of the mixer 26, creates an effect sound 

35 in addition to the back music by the software synthesiz- 
er, and outputs these onto the speaker 19 through the 
mixer 15 and output buffer 17. As a result, an effect 
sound due to MIDI (e.g. explosion sound) as in (b) is 
outputted from the speaker 1 9. During ringing of this ef- 

40 feet sound, the back music continuously sounds as in 
(c) . Then, if the effect-sound signal ends, the effect 
sound from the speaker 1 9 ceases resulting in sounding 
of the back music only. Incidentally, the game image is 
continuously displayed on the display 20 as in (d). In this 

45 manner, according to the data reproducing apparatus of 
Fig. 30, a game machine can be realized which is capa- 
ble of ringing an effect sound due to MIDI over a back 
music. Also, because the software synthesizer of the 
MIDI reproducing section 11 can be commonly used in 

50 creating an effect sound, the process is simplified. 

[0148] The use of the data reproducing apparatus of 
the invention can realize a system having various func- 
tions besides the above. Fig. 32 to Fig. 34 are one ex- 
ample of the same, showing an example that a given 

55 benefit is provided to the persons who have viewed a 
particular CM on the Internet. Each of data of MIDI, au- 
dio, text and images is mixed in CM information in a 
chronological fashion, as in Fig. 33. Consequently, a tag 
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describing URL (Uniform Resource Locator) as shown 
in Fig. 34 is inserted in the last part of text data (broken 
lines Z). In the tag, "XXX" in the last is information rep- 
resentative of what CM it is. 

[0149] Explaining according to a flowchart of Fig. 32, 
the viewer first downloads CM data from a file 1 a (see 
Fig. 1 , Fig. 1 9) in a server on the Internet (S601 ). This 
CM data is received by the data receiving section 3, as- 
sorted to each section by the data assorting section 4, 
and reproduced by the foregoing procedure and output- 
ted from the speaker 19 and display 20. Herein, if the 
text data received is reproduced to the last by the text 
reproducing section 13, a tag shown in 34 is read out 
(S602). 

[0150] Subsequently, a browser (perusal software) is 
started up (S603), and jump is made to a homepage of 
a URL described in the read-out tag (S604). The server 
jumped to (not shown) interprets the "XXX" part of the 
tag to determine as to what CM has been viewed (S605), 
and in case that a product of the CM is purchased over 
the network, to carry out a process, for example, of 
charge at a rate discounted by 20% (S606). Thus, ac- 
cording to the above system, discount service can be 
offered to the persons who have viewed the CM. 
[0151] Fig. 35 and Fig. 36 is another application ex- 
ample using the data reproducing apparatus of the in- 
vention, showing an example of providing ticket dis- 
count service to a person who has purchased music da- 
ta over the Internet. In this case, the music data is added 
with words, music commentary, introduction of players 
or the like as text data, wherein a tag as shown in Fig. 
36 is inserted in the last part of the text data. In the tag, 
"from = 2000/08/1 5 to = 2000/09/1 5" represents that the 
ticket available term is from August 1 5, 2000 to Septem- 
ber 15, 2000. Meanwhile, "YYY" in the last is the infor- 
mation representative of what the purchased music data 
is. 

[0152] Explaining according to a flowchart of Fig. 35, 
the viewer first downloads music data from a file 1a in 
a server on the Internet (S701). The music data is re- 
ceived by the data receiving section 3 and assorted by 
the data assorting section 4 to each section, followed by 
being reproduced in the foregoing procedure and out- 
putted from the speaker 1 9 and display 20. Also, each 
of data is stored to and saved in the external storage 
device 22 (in Fig. 37, information storage medium 70). 
Herein, if the received text data is reproduced to the last 
in the text reproducing section 13, the tag shown in Fig. 
36 is read out (S702). 

[0153] Subsequently, a browser is started up (S703), 
to determine the current date is within the available term 
or not (S704). This determination is made by making ref- 
erence to the available term described in the foregoing 
tag. If within the available term (S704 YES), jump is 
made to a homepage of the URL described in the read- 
out tag (S705). If not within the available term (S704 
NO), nothing is done for end (S708). 
[0154] The server jumped to (not shown) interprets 



the "YYY" part of the tag to determine what music data 
has been purchased (S706), and transmits aguide mes- 
sage that a ticket of a concert by the musical artist is to 
be purchased at a discount price thus displaying the 
5 message on the display 20 (S707). Therefore, accord- 
ing to the above system, it is possible to induce the per- 
son who has purchased music data to purchase a ticket. 

INDUSTRIAL APPLICABILITY 

10 

[01 55] The data reproducing apparatus of the present 
invention can be mounted on an information terminal in 
various kinds, such as a personal computer and an In- 
ternet-TV STB (Set Top Box) besides the foregoing cel- 
15 lular phone or game machine. 



Claims 

20 1. A data reproducing apparatus for receiving and re- 
producing data including event information and time 
information for executing an event, the data repro- 
ducing apparatus comprising: 

25 a data receiving section capable of receiving a 

plurality of kinds of data having event informa- 
tion different in attribute; 
a data assorting section for assorting data on a 
kind-by-kind basis depending upon time infor- 
30 mation of each of data received by said data 

receiving section; 

a data reproducing section for reproducing the 
data assorted by said data assorting section; 
and 

35 an output section for outputting data repro- 

duced by said data reproducing section. 

2. A data reproducing apparatus according to claim 1 , 
wherein the plurality of kinds of data comprise first 

40 data having MIDI event information and second da- 
ta having event information of other than MIDI. 

3. A data reproducing apparatus according to claim 2, 
wherein the second data includes data having text 

45 event information and data having image event in- 
formation. 

4. A data reproducing apparatus according to claim 3, 
wherein the second data further includes data hav- 

50 ing audio event information. 

5. A data reproducing apparatus according to claim 2, 
wherein the first and second data comprises data 
in an SMF type, the second data having an extend- 

55 ed format, and data to be reproduced being record- 
ed in event information of the extended format. 

6. A data reproducing apparatus for receiving and re- 
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producing data including event information and time 
information for executing an event, the data repro- 
ducing apparatus comprising: 

a data receiving section capable of receiving 
data having MIDI event information, data hav- 
ing text event information and data having im- 
age event information; 

a data assorting section for assorting data on a 
kind-by-kind basis depending upon time infor- 
mation of each of data received by said data 
receiving section; 

a data reproducing section for executing an 
event recorded in data assorted by said data 
assorting section thereby reproducing the data; 
a first output section for outputting, as a sound, 
MIDI data reproduced by said data reproducing 
section; and 

a second output section for outputting, as visi- 
ble information, text and image data repro- 
duced in said data reproducing section. 

7. A data reproducing apparatus according to claim 6, 
wherein said data receiving section is further capa- 
ble of receiving data having audio event informa- 
tion, said first output section outputting, as a sound, 
MIDI and audio data reproduced by said data repro- 
ducing section. 

8. A data reproducing apparatus according to claim 7, 
having 

a first mixer for mixing MIDI and audio data re- 
produced by said data reproducing section, and 
a second mixer for mixing text and image data 
reproduced by said data reproducing section, 
wherein 

said first output section outputs data mixed by 
said first mixer, 

said second output section outputting data 
mixed by said second mixer. 

9. A data reproducing method for receiving and repro- 
ducing data including event information and time in- 
formation for executing an event, the data reproduc- 
ing method comprising: 

a step of receiving first data having MIDI event 
information and second data having event in- 
formation of other than MIDI; 
a step of assorting data on a kind-by-kind basis 
depending upon time information of each of re- 
ceived data; 

a step of reproducing assorted data; and 
a step of outputting reproduced data. 

10. A data reproducing method for reproducing the sec- 
ond data repeatedly by a predetermined number of 



times according to claim 9, the data reproducing 
method including: 

a step of, upon receiving first the second data, 
5 storing reproduced data recorded in the second 

data to a memory; and 

a step of, upon repetitively reproducing the sec- 
ond data, reading, out of said memory, and re- 
producing the reproduced data according to 
10 time information of the second data. 

11. A data reproducing method having the reproduced 
data recorded in the second data wholly or partly 
divided into a plurality of data to reproduce the sec- 
15 ond data following the first data, according to claim 
9, the data reproducing method including: 

a step of receiving a data group having a plu- 
rality of division data inserted between preced- 
20 ing ones of the first data, to extract inserted di- 

vision data from the data group; and 
a step of combining extracted division data into 
reproduced data. 

25 12. A data reproducing method according to claim 11, 
wherein the division data is sequentially stored to 
said memory in a chronological fashion, a start ad- 
dress of division data to be coupled to the division 
data being recorded in an area of stored division 
30 data. 

13. A data reproducing method according to claim 9, 
wherein reproduced data to be recorded in the sec- 
ond data is cut of a silent section having a signal 

35 level not exceeding a constant value. 

14. A data reproducing method according to claim 13, 
wherein a fade-in/out process is done in a signal in 
a vicinity of a rise portion and fall portion of the re- 

40 produced data. 

15. A data reproducing apparatus for receiving and re- 
producing data including event information and time 
information for executing an event, the data repro- 

45 ducing apparatus comprising: 

a data receiving section capable of receiving a 
plurality of kinds of data having event informa- 
tion different in attribute; 
50 a data assorting section for assorting data to be 

processed within a unit section having a prede- 
termined time width, for each unit section, on a 
kind-by-kind basis depending upon time infor- 
mation of each of data received by said data 
55 receiving section; 

a storage section for temporarily storing data 
assorted by said data assorting section on a 
kind-by-kind basis; 
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a data reproducing section for sequentially 
reading out, in a next unit section, data at each 
unit section stored in said storage section and 
executing an event recorded in each of data 
thereby reproducing the data; and 
an output section for outputting data repro- 
duced by said data reproducing section. 

16. A data reproducing apparatus according to claim 

15, wherein said data assorting section assorts, for 
storage to said storage section, the data to be proc- 
essed on a kind-by-kind basis in a last timing of a 
unit section, 

said data reproducing section sequentially 
reading out, in a next unit section, data at a unit sec- 
tion assorted by said data assorting section to exe- 
cute an event of the relevant data. 

17. A data reproducing apparatus according to claim 

1 6, wherein the time information is a delta time de- 
fined as a time of from an execution time point of a 
last-time event to an execution of an event in this 
time, 

said data assorting section calculating a time 
width of a process section that data in this time 
is to be processed from a difference between a 
time at present as a time at a last in a unit sec- 
tion and an execution time of a last event in a 
unit section precedent by one, and assorting 
and storing, into storage section, unit-section 
data such that a sum of a delta time of each 
event in the relevant process section is fallen 
within a range of the time width, 
said data reproducing section reproducing unit- 
section data assorted by said data assorting 
section in a next unit section having a same 
time width as the relevant unit section. 

18. A data reproducing apparatus according to any of 
claims 1 5 to 1 7, wherein provided is a timing control 
section for controlling timings at a start and end of 
the unit section. 

19. A data reproducing apparatus according to claim 
18, wherein said output section has a function to 
count the number of output data to forward a control 
signal to said timing control section depending upon 
a count value thereof, said timing control section 
outputting a timing signal depending upon the con- 
trol signal. 

20. A data reproducing method for receiving and repro- 
ducing data including event information and time in- 
formation for executing an event, the data reproduc- 
ing method comprising: 



having event information different in attribute; 
a step of assorting, and temporarily storing to 
a storage section, data to be processed within 
a unit section having a predetermined time 
5 width on a kind-by-kind basis, for each unit sec- 

tion, depending upon time information of each 
of data received; 

a step of sequentially reading out, in a next unit 
section, data at each unit section stored in said 
10 storage section and executing an event record- 

ed in the relevant data thereby reproducing the 
data; and 

a step of outputting reproduced data. 

15 21. A data reproducing apparatus for reproducing, 
while downloading, stream data according to claim 
1 or 15, wherein 

said data receiving section has a buffer, 
20 said data receiving section calculating a data 

transfer capacity J per unit time and a data con- 
sumption amount E per unit time on the basis 
of data first received by said data receiving sec- 
tion, 

25 where J < E, reproducing is started after cach- 

ing an required amount of data being to said 
buffer, while where J > E, reproducing being 
made while intermittently receiving data without 
carrying out caching. 

30 

22. A information terminal mounted with a data repro- 
ducing apparatus according to claim 1 or 1 5, where- 
in various kinds of data are to be downloaded, the 
information terminal including a sound generating 
35 section for outputting sound depending upon down- 
loaded data and a display for displaying a text and 
image depending upon the downloaded data. 



23. An information terminal according to claim 22, 
40 which is an information terminal having a function 

of a phone, wherein, when said data receiving sec- 
tion receives an incoming-call signal in a state of 
outputting a sound, the sound is prohibited from be- 
ing outputted thereby outputting an incoming tone. 

45 

24. An information terminal according to claim 22, 
which is an information terminal having a function 
of a game machine, wherein, when said data receiv- 
ing section receives an effect-sound signal in a 

50 state of outputting a sound, an effect sound is out- 
putted together with the sound. 

25. An information terminal according to claim 22, 
wherein music data due to MIDI, music words data 

55 due to text and jacket picture data due to images 
are to be downloaded. 
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a step of receiving a plurality of kinds of data 26. An information terminal according to any of claims 
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22 to 25, wherein a small-sized information storage 
medium is to be attached and detached, to store 
each of data downloaded to said information stor- 
age medium. 

5 

27. A data reproducing apparatus according to claim 3, 
wherein commercial information including text is to 
be received, the text data including an URL to be 
jumped to upon starting up a browser of the Internet 
and information concerning service to be offered in 10 
the URL 
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FIG. 7 
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FIG. 10 
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